Method and apparatus for the visualization of three-dimensional objects

ABSTRACT

A computer-implemented method and a corresponding apparatus are provided for the provision of a two-dimensional visualization image having a plurality of visualization pixels for the visualization of a three-dimensional object represented by volume data for a user. Context information for the visualization is obtained by the evaluation of natural language and is taken into account in the visualization. The natural language can be in the form of electronic documents, which are assigned or can be assigned to the visualization process. In addition, the natural language can be in the form of a speech input of a user, during or after the visualization.

PRIORITY STATEMENT

The present application hereby claims priority under 35 U.S.C. § 119 toGerman patent application number DE 102020213234.8 filed Oct. 20, 2020,the entire contents of which are hereby incorporated herein byreference.

FIELD

Example embodiments of the invention generally lie in the field ofvolume rendering, in other words the representation or visualization ofthree-dimensional bodies or objects.

BACKGROUND

The modeling, reconstruction or visualization of three-dimensionalobjects has a wide range of application in the fields of medicine (forexample CT, PET), physics (for example electron structure of largemolecules) or geophysics (composition and position of layers of earth).Typically, the object to be examined is irradiated (for example by wayof electromagnetic waves or soundwaves) in order to examine itscomposition. The scattered radiation is detected and properties of thebody are determined from the detected values. Conventionally, the resultconsists in a physical variable (for example density, tissue type,elasticity, speed), whose value is determined for the body. As a rule, avirtual grid is used in this case at whose grid points the value of thevariable is determined. These grid points are conventionally referred toas voxels. The term “voxel” is a portmanteau formed from the terms“volume” and “pixel”. A voxel corresponds to the spatial coordinate of agrid point, to which the value of a variable at this location isassigned. This is usually a physical variable, which can be representedas a scalar or vectorial field, in other words the corresponding fieldvalue is assigned to the spatial coordinate. The value of the variableor of the field at any location (in other words at any object points ofthe examined object) can be obtained by interpolation of the voxels.

A three-dimensional representation of the examined object or body isgenerated from the voxels on a two-dimensional display area (for examplea screen or a panel or lens of what are known as “augmented realityglasses”) for visualization of the volume data. In other words, voxels(defined in three dimensions) are mapped onto pixels (defined in twodimensions) of a two-dimensional visualization image. The pixels of thevisualization image will also be called visualization pixels below. Themapping is conventionally referred to as volume rendering. Theimplementation of the volume rendering determines how informationcontained in the voxels is rendered by way of the pixels.

One of the most used methods of volume rendering is what is known as raycasting (cf. Levoy: “Display of Surfaces from Volume Data”, IEEEcomputer Graphics and Applications, issue 8, no. 3, May 1988, pages29-37). In ray casting, simulated rays, which emanate from the eye of animaginary observer, are sent through the examined body or the examinedobject. Along the rays, RGBA values are determined from the voxels forsampling points and are combined by way of Alpha Compositing or AlphaBlending to form pixels for a two-dimensional image. In the expressionRGBA, the letters R, G and B stand for the color components red, greenand blue of which the color contribution of the corresponding samplingpoint is composed. A stands for the ALPHA value, which is a measure ofthe transparency at the sampling point. The respective transparency isused in the overlaying of RGB values at sampling points relative to thepixel. Lighting effects are conventionally taken into account by way ofa lighting model in the context of a method designated “Shading”.

A further method of volume rendering is what is known as path tracing(cf. Kajiya: “The rendering equation”, ACM SIGGRAPH computer Graphics,issue 20, no. 4, August 1986, pages 143-150). In this case, a pluralityof simulated rays are shot into the volume data per visualization pixel,and these then interact with the volume, in other words are reflected,refracted or absorbed, wherein every time (apart from in the case ofabsorption) at least one random ray is generated. Each simulated raythus looks for its path through the volume data. The more virtual raysused per visualization pixel, the more the ideal image is approached. Inparticular the processes and methods described in EP 3 178 068 B1 can beapplied here. The disclosure of EP 3 178 068 B1 is hereby incorporatedherein, in its entirety, by way of reference.

Users primarily in the medical and clinical sector can be effectivelysupported by visualization methods of this kind since a representationof this kind allows a rapid overview of complex anatomicalrelationships. For example, this allows for better planning of surgicalinterventions. Furthermore, such visualization images enable thecreation of meaningful medical reports.

SUMMARY

The inventors have discovered that a primary technical obstacle inimplementation of a system for interactive volume rendering is thecomplex operation of such systems. In order to generate a visualizationimage optimally adjusted to the situation, many parameters of such asystem have to be set and optimized. For example, particular anatomiesare highly relevant to different medical issues, while other anatomiesare irrelevant and obstruct the view of what is essential.

The inventors have discovered that a large number of parameters has tobe purposefully adjusted in the case of volume rendering algorithms inorder to work out such aspects. Thus, for example, particular anatomiescan be represented transparently while other anatomies can be optimallyilluminated. In order to make suitable adjustments, a user requires acertain amount of time, which is often not available in daily clinicalpractice, in addition to some background knowledge in respect of themode of operation of the respective volume rendering algorithm. What ismore, the inventors have discovered that adversely selectedvisualization parameters can obstruct pathologically relevant issues andlead to incorrect decisions.

It is an aim of at least one embodiment of the present invention,therefore to provide methods and apparatuses for the visualization ofvolume data that are improved in this respect. In at least oneembodiment, this should address the problem of providing visualizations,which are better adapted to the underlying problem and, moreover, can beadapted by a user in as simple and intuitively operable a manner aspossible.

At least one embodiment of the invention is directed to a method, anapparatus, a computer program product or a computer-readable storagemedium. Advantageous developments are disclosed in the claims.

The inventive solution of embodiments will be described below inrelation to the apparatuses as well as in relation to the method.Features, advantages or alternative embodiments mentioned in thisconnection should likewise also be transferred to the other subjectmatters, and vice versa. In other words, the concrete claims (which aredirected, for example, to a device) can also be developed with thefeatures, which are described or claimed in connection with a method.The corresponding functional features of the method are formed bycorresponding concrete modules.

Furthermore, embodiments will be described in relation to methods andapparatuses for the visualization of a three-dimensional body as well asin relation to methods and apparatuses for adjusting trained functions.Features and alternative embodiments of data structures and/or functionsin methods and apparatuses for determination can be transferred here toanalogous data structures and/or functions in methods and apparatusesfor adjusting. Analogous data structures can be identified here inparticular by the use of the prefix “training”. Furthermore, the trainedfunctions used in methods and apparatuses for the visualization of athree-dimensional body can have been adjusted and/or provided inparticular by methods and apparatuses for adjusting trained functions.

According to at least one embodiment, a computer-implemented method forthe provision of a two-dimensional visualization image having aplurality of visualization pixels for the visualization of athree-dimensional object represented by volume data is provided for auser. The method comprises:

providing context data containing natural language and/or text inrespect of the visualization of the three-dimensional object;

determining a mapping indication based on the context data, wherein themapping indication has information about a mapping of the volume dataonto the visualization image that is suitable based on the context data;

determining a mapping rule based on the mapping indication;

mapping the volume data onto the visualization pixels using the mappingrule for the creation of the visualization image; and

providing the visualization image.

According to a further embodiment, a computer-implemented method forproviding a trained function for determining a mapping rule isdisclosed. The method comprises:

providing training input data, wherein the training input data hascontext data containing natural language and/or text in respect of thevisualization of a three-dimensional object and volume data representingthe three-dimensional object;

providing training output data, wherein the training output data has amapping rule for the generation of a suitable two-dimensionalvisualization image by mapping the volume data;

generating a training mapping rule by applying the trained function tothe training input data;

comparing the training mapping rule with the training output data;

adjusting the trained function based upon the comparison.

According to a further embodiment, an apparatus is disclosed for theprovision of a two-dimensional visualization image having a plurality ofvisualization pixels for visualization of a three-dimensional objectrepresented by volume data for a user, which apparatus comprises:

an interface, which is designed to receive the volume data and contextdata containing natural language in respect of the visualization of thethree-dimensional object;

a configuration module, which is designed to:

-   -   determine a mapping indication based on the context data (and        optionally on the volume data), wherein the mapping indication        comprises information about a mapping of the volume data, which        is suitable based on the context data, onto the visualization        image; and    -   determine a mapping rule based on the mapping indication; and

a visualization module, which is designed to:

-   -   map the volume data onto the visualization pixels using the        mapping rule for the creation of the visualization image; and    -   provide the visualization image.

The configuration module and/or the visualization module of anembodiment can be part of an computing facility. The configurationmodule can be designed to carry out the configuration algorithm. Thevisualization module can be designed to carry out the image synthesisalgorithm.

The computing facility can be designed as a central or decentralcomputing facility. The computing facility can have one or moreprocessor(s). The processors can be designed as a central processingunit (CPU for short) and/or as graphics processing unit (GPU for short).The computing facility can be designed as what is known as asystem-on-a-chip (SoP for short), which controls all functions of adevice. Alternatively, the computing facility can be implemented as alocal or Cloud-based processing server.

The invention relates in a further embodiment to a computer programproduct, which comprises a program and can be loaded directly into astorage device of a programmable computing facility and has programcode/segments, for example libraries and auxiliary functions, in orderto carry out an embodiment of a method for the visualization of athree-dimensional object in particular according to the embodiment whenthe computer program product is run.

Furthermore, the invention relates in a further embodiment to a computerprogram product, which comprises a program and direct can be loadeddirectly into a storage device of a programmable computing facility andhas program code/segments, for example libraries and auxiliaryfunctions, in order to carry out a method for the provision of a trainedfunction in particular according to the embodiment when the computerprogram product is run.

The computer program products of an embodiment can comprise softwarewith a source code, which still has to be compiled and linked or whichonly has to be interpreted, or an executable software code, which forexecution only has to be loaded into the processing unit. As a result ofthe computer program products the method can be carried out quickly,repeatedly in an identical manner and robustly. The computer programproducts are configured such that they can carry out the inventivemethod steps by way of the computing facility. The computing facilityhas to have in each case the requirements such as an appropriate mainmemory, an appropriate processor, an appropriate graphics card or anappropriate logic unit, so the respective method steps of an embodimentcan be efficiently carried out.

The computer program products are stored, for example in an embodiment,on a computer-readable storage medium or saved on a network or serverfrom where they can be loaded into the processor of the respectivecomputing facility, which is directly connected to the computingfacility or can be designed as part of the computing facility.Furthermore, control information of the computer program products can bestored on a computer-readable storage medium. The control information ofthe computer-readable storage medium can be configured in such a waythat it carries out an inventive method when the data carrier is used inan computing facility. Examples of computer-readable storage medium area DVD, a magnetic tape or a USB stick on which electronically readablecontrol information, in particular software, is stored. When thiscontrol information is read from the data carrier and stored in ancomputing facility, all inventive embodiments of the previouslydescribed method can be carried out. The invention can thus also startfrom the computer-readable medium and/or the computer-readable storagemedium. The advantages of the proposed computer program product or theassociated computer-readable media substantially match the advantages ofan embodiment of the proposed method.

At least one embodiment of the invention is directed to acomputer-implemented method for creating a two-dimensional visualizationimage including a plurality of visualization pixels for visualization ofa three-dimensional object represented by volume data for a user, themethod comprising:

providing context data containing at least one of natural language andtext for the visualization of the three-dimensional object;

determining a mapping rule based on the volume data and the contextdata; and

mapping the volume data onto the plurality of visualization pixels usingthe mapping rule, for creation of the two-dimensional visualizationimage.

At least one embodiment of the invention is directed to an apparatus forproviding a two-dimensional visualization image including a plurality ofvisualization pixels for visualization of a three-dimensional objectrepresented by volume data for a user, comprising:

an interface, designed to receive the volume data and at least one ofnatural language and context data containing text relating to thevisualization of the three-dimensional object;

a computing facility, designed to determine a mapping rule based on thevolume data and the context data; and

a visualization module, designed to

-   -   map the volume data onto the plurality of visualization pixels        using the mapping rule, to create the two-dimensional        visualization image; and    -   provide the visualization image.

At least one embodiment of the invention is directed to a non-transitorycomputer program product storing a computer program, directly loadableinto a storage device of a computing facility, including programsegments to carry out the method of an embodiment when the programsegments are run in the computing facility.

At least one embodiment of the invention is directed to a non-transitorycomputer-readable storage medium storing program segments, readable andrunnable by a computing facility, to carry out the method of anembodiment when the program segments are run in the computing facility.

At least one embodiment of the invention is directed to an apparatus forproviding a two-dimensional visualization image including a plurality ofvisualization pixels for visualization of a three-dimensional objectrepresented by volume data for a user, comprising:

an interface to receive the volume data and at least one of naturallanguage and context data containing text relating to the visualizationof the three-dimensional object;

at least one processor to determine a mapping rule based on the volumedata and the context data to map the volume data onto the plurality ofvisualization pixels using the mapping rule, to create thetwo-dimensional visualization image.

At least one embodiment of the invention is directed to an apparatusfurther comprising a display to display the two-dimensionalvisualization image.

BRIEF DESCRIPTION OF THE DRAWINGS

Further particulars and advantages of the invention will become obviousfrom the following explanations of example embodiments with reference toschematic drawings. Modifications mentioned in this connection can becombined with one another in each case in order to form new embodiments.Identical reference numerals will be used in different figures foridentical features.

In the drawings:

FIG. 1 shows a schematic representation of a first embodiment of asystem/method for the visualization of a three-dimensional object,

FIG. 2 shows a schematic representation of a further embodiment of asystem for the visualization of a three-dimensional object,

FIG. 3 shows a flowchart of a method for the visualization of athree-dimensional object according to one embodiment,

FIG. 4 shows a trained function for generating an improved visualizationimage,

FIG. 5 shows a schematic representation of an embodiment of a system forproviding the trained function, and

FIG. 6 shows a flowchart of a method for the provision of a trainedfunction for improvement of a visualization image of a three-dimensionalobject according to one embodiment.

DETAILED DESCRIPTION OF THE EXAMPLE EMBODIMENTS

The drawings are to be regarded as being schematic representations andelements illustrated in the drawings are not necessarily shown to scale.Rather, the various elements are represented such that their functionand general purpose become apparent to a person skilled in the art. Anyconnection or coupling between functional blocks, devices, components,or other physical or functional units shown in the drawings or describedherein may also be implemented by an indirect connection or coupling. Acoupling between components may also be established over a wirelessconnection. Functional blocks may be implemented in hardware, firmware,software, or a combination thereof.

Various example embodiments will now be described more fully withreference to the accompanying drawings in which only some exampleembodiments are shown. Specific structural and functional detailsdisclosed herein are merely representative for purposes of describingexample embodiments. Example embodiments, however, may be embodied invarious different forms, and should not be construed as being limited toonly the illustrated embodiments. Rather, the illustrated embodimentsare provided as examples so that this disclosure will be thorough andcomplete, and will fully convey the concepts of this disclosure to thoseskilled in the art. Accordingly, known processes, elements, andtechniques, may not be described with respect to some exampleembodiments. Unless otherwise noted, like reference characters denotelike elements throughout the attached drawings and written description,and thus descriptions will not be repeated. At least one embodiment ofthe present invention, however, may be embodied in many alternate formsand should not be construed as limited to only the example embodimentsset forth herein.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, components, regions,layers, and/or sections, these elements, components, regions, layers,and/or sections, should not be limited by these terms. These terms areonly used to distinguish one element from another. For example, a firstelement could be termed a second element, and, similarly, a secondelement could be termed a first element, without departing from thescope of example embodiments of the present invention. As used herein,the term “and/or,” includes any and all combinations of one or more ofthe associated listed items. The phrase “at least one of” has the samemeaning as “and/or”.

Spatially relative terms, such as “beneath,” “below,” “lower,” “under,”“above,” “upper,” and the like, may be used herein for ease ofdescription to describe one element or feature's relationship to anotherelement(s) or feature(s) as illustrated in the figures. It will beunderstood that the spatially relative terms are intended to encompassdifferent orientations of the device in use or operation in addition tothe orientation depicted in the figures. For example, if the device inthe figures is turned over, elements described as “below,” “beneath,” or“under,” other elements or features would then be oriented “above” theother elements or features. Thus, the example terms “below” and “under”may encompass both an orientation of above and below. The device may beotherwise oriented (rotated 90 degrees or at other orientations) and thespatially relative descriptors used herein interpreted accordingly. Inaddition, when an element is referred to as being “between” twoelements, the element may be the only element between the two elements,or one or more other intervening elements may be present.

Spatial and functional relationships between elements (for example,between modules) are described using various terms, including“connected,” “engaged,” “interfaced,” and “coupled.” Unless explicitlydescribed as being “direct,” when a relationship between first andsecond elements is described in the above disclosure, that relationshipencompasses a direct relationship where no other intervening elementsare present between the first and second elements, and also an indirectrelationship where one or more intervening elements are present (eitherspatially or functionally) between the first and second elements. Incontrast, when an element is referred to as being “directly” connected,engaged, interfaced, or coupled to another element, there are nointervening elements present. Other words used to describe therelationship between elements should be interpreted in a like fashion(e.g., “between,” versus “directly between,” “adjacent,” versus“directly adjacent,” etc.).

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of exampleembodiments of the invention. As used herein, the singular forms “a,”“an,” and “the,” are intended to include the plural forms as well,unless the context clearly indicates otherwise. As used herein, theterms “and/or” and “at least one of” include any and all combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “comprises,” “comprising,” “includes,” and/or“including,” when used herein, specify the presence of stated features,integers, steps, operations, elements, and/or components, but do notpreclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and/or groupsthereof. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items. Expressionssuch as “at least one of,” when preceding a list of elements, modify theentire list of elements and do not modify the individual elements of thelist. Also, the term “example” is intended to refer to an example orillustration.

When an element is referred to as being “on,” “connected to,” “coupledto,” or “adjacent to,” another element, the element may be directly on,connected to, coupled to, or adjacent to, the other element, or one ormore other intervening elements may be present. In contrast, when anelement is referred to as being “directly on,” “directly connected to,”“directly coupled to,” or “immediately adjacent to,” another elementthere are no intervening elements present.

It should also be noted that in some alternative implementations, thefunctions/acts noted may occur out of the order noted in the figures.For example, two figures shown in succession may in fact be executedsubstantially concurrently or may sometimes be executed in the reverseorder, depending upon the functionality/acts involved.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which example embodiments belong. Itwill be further understood that terms, e.g., those defined in commonlyused dictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

Before discussing example embodiments in more detail, it is noted thatsome example embodiments may be described with reference to acts andsymbolic representations of operations (e.g., in the form of flowcharts, flow diagrams, data flow diagrams, structure diagrams, blockdiagrams, etc.) that may be implemented in conjunction with units and/ordevices discussed in more detail below. Although discussed in aparticularly manner, a function or operation specified in a specificblock may be performed differently from the flow specified in aflowchart, flow diagram, etc. For example, functions or operationsillustrated as being performed serially in two consecutive blocks mayactually be performed simultaneously, or in some cases be performed inreverse order. Although the flowcharts describe the operations assequential processes, many of the operations may be performed inparallel, concurrently or simultaneously. In addition, the order ofoperations may be re-arranged. The processes may be terminated whentheir operations are completed, but may also have additional steps notincluded in the figure. The processes may correspond to methods,functions, procedures, subroutines, subprograms, etc.

Specific structural and functional details disclosed herein are merelyrepresentative for purposes of describing example embodiments of thepresent invention. This invention may, however, be embodied in manyalternate forms and should not be construed as limited to only theembodiments set forth herein.

Units and/or devices according to one or more example embodiments may beimplemented using hardware, software, and/or a combination thereof. Forexample, hardware devices may be implemented using processing circuitysuch as, but not limited to, a processor, Central Processing Unit (CPU),a controller, an arithmetic logic unit (ALU), a digital signalprocessor, a microcomputer, a field programmable gate array (FPGA), aSystem-on-Chip (SoC), a programmable logic unit, a microprocessor, orany other device capable of responding to and executing instructions ina defined manner. Portions of the example embodiments and correspondingdetailed description may be presented in terms of software, oralgorithms and symbolic representations of operation on data bits withina computer memory. These descriptions and representations are the onesby which those of ordinary skill in the art effectively convey thesubstance of their work to others of ordinary skill in the art. Analgorithm, as the term is used here, and as it is used generally, isconceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of optical, electrical, or magnetic signals capable of beingstored, transferred, combined, compared, and otherwise manipulated. Ithas proven convenient at times, principally for reasons of common usage,to refer to these signals as bits, values, elements, symbols,characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise, or as is apparent from the discussion,terms such as “processing” or “computing” or “calculating” or“determining” of “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computingdevice/hardware, that manipulates and transforms data represented asphysical, electronic quantities within the computer system's registersand memories into other data similarly represented as physicalquantities within the computer system memories or registers or othersuch information storage, transmission or display devices.

In this application, including the definitions below, the term ‘module’or the term ‘controller’ may be replaced with the term ‘circuit.’ Theterm ‘module’ may refer to, be part of, or include processor hardware(shared, dedicated, or group) that executes code and memory hardware(shared, dedicated, or group) that stores code executed by the processorhardware.

The module may include one or more interface circuits. In some examples,the interface circuits may include wired or wireless interfaces that areconnected to a local area network (LAN), the Internet, a wide areanetwork (WAN), or combinations thereof. The functionality of any givenmodule of the present disclosure may be distributed among multiplemodules that are connected via interface circuits. For example, multiplemodules may allow load balancing. In a further example, a server (alsoknown as remote, or cloud) module may accomplish some functionality onbehalf of a client module.

Software may include a computer program, program code, instructions, orsome combination thereof, for independently or collectively instructingor configuring a hardware device to operate as desired. The computerprogram and/or program code may include program or computer-readableinstructions, software components, software modules, data files, datastructures, and/or the like, capable of being implemented by one or morehardware devices, such as one or more of the hardware devices mentionedabove. Examples of program code include both machine code produced by acompiler and higher level program code that is executed using aninterpreter.

For example, when a hardware device is a computer processing device(e.g., a processor, Central Processing Unit (CPU), a controller, anarithmetic logic unit (ALU), a digital signal processor, amicrocomputer, a microprocessor, etc.), the computer processing devicemay be configured to carry out program code by performing arithmetical,logical, and input/output operations, according to the program code.Once the program code is loaded into a computer processing device, thecomputer processing device may be programmed to perform the programcode, thereby transforming the computer processing device into a specialpurpose computer processing device. In a more specific example, when theprogram code is loaded into a processor, the processor becomesprogrammed to perform the program code and operations correspondingthereto, thereby transforming the processor into a special purposeprocessor.

Software and/or data may be embodied permanently or temporarily in anytype of machine, component, physical or virtual equipment, or computerstorage medium or device, capable of providing instructions or data to,or being interpreted by, a hardware device. The software also may bedistributed over network coupled computer systems so that the softwareis stored and executed in a distributed fashion. In particular, forexample, software and data may be stored by one or more computerreadable recording mediums, including the tangible or non-transitorycomputer-readable storage media discussed herein.

Even further, any of the disclosed methods may be embodied in the formof a program or software. The program or software may be stored on anon-transitory computer readable medium and is adapted to perform anyone of the aforementioned methods when run on a computer device (adevice including a processor). Thus, the non-transitory, tangiblecomputer readable medium, is adapted to store information and is adaptedto interact with a data processing facility or computer device toexecute the program of any of the above mentioned embodiments and/or toperform the method of any of the above mentioned embodiments.

Example embodiments may be described with reference to acts and symbolicrepresentations of operations (e.g., in the form of flow charts, flowdiagrams, data flow diagrams, structure diagrams, block diagrams, etc.)that may be implemented in conjunction with units and/or devicesdiscussed in more detail below. Although discussed in a particularlymanner, a function or operation specified in a specific block may beperformed differently from the flow specified in a flowchart, flowdiagram, etc. For example, functions or operations illustrated as beingperformed serially in two consecutive blocks may actually be performedsimultaneously, or in some cases be performed in reverse order.

According to one or more example embodiments, computer processingdevices may be described as including various functional units thatperform various operations and/or functions to increase the clarity ofthe description. However, computer processing devices are not intendedto be limited to these functional units. For example, in one or moreexample embodiments, the various operations and/or functions of thefunctional units may be performed by other ones of the functional units.Further, the computer processing devices may perform the operationsand/or functions of the various functional units without sub-dividingthe operations and/or functions of the computer processing units intothese various functional units.

Units and/or devices according to one or more example embodiments mayalso include one or more storage devices. The one or more storagedevices may be tangible or non-transitory computer-readable storagemedia, such as random access memory (RAM), read only memory (ROM), apermanent mass storage device (such as a disk drive), solid state (e.g.,NAND flash) device, and/or any other like data storage mechanism capableof storing and recording data. The one or more storage devices may beconfigured to store computer programs, program code, instructions, orsome combination thereof, for one or more operating systems and/or forimplementing the example embodiments described herein. The computerprograms, program code, instructions, or some combination thereof, mayalso be loaded from a separate computer readable storage medium into theone or more storage devices and/or one or more computer processingdevices using a drive mechanism. Such separate computer readable storagemedium may include a Universal Serial Bus (USB) flash drive, a memorystick, a Blu-ray/DVD/CD-ROM drive, a memory card, and/or other likecomputer readable storage media. The computer programs, program code,instructions, or some combination thereof, may be loaded into the one ormore storage devices and/or the one or more computer processing devicesfrom a remote data storage device via a network interface, rather thanvia a local computer readable storage medium. Additionally, the computerprograms, program code, instructions, or some combination thereof, maybe loaded into the one or more storage devices and/or the one or moreprocessors from a remote computing system that is configured to transferand/or distribute the computer programs, program code, instructions, orsome combination thereof, over a network. The remote computing systemmay transfer and/or distribute the computer programs, program code,instructions, or some combination thereof, via a wired interface, an airinterface, and/or any other like medium.

The one or more hardware devices, the one or more storage devices,and/or the computer programs, program code, instructions, or somecombination thereof, may be specially designed and constructed for thepurposes of the example embodiments, or they may be known devices thatare altered and/or modified for the purposes of example embodiments.

A hardware device, such as a computer processing device, may run anoperating system (OS) and one or more software applications that run onthe OS. The computer processing device also may access, store,manipulate, process, and create data in response to execution of thesoftware. For simplicity, one or more example embodiments may beexemplified as a computer processing device or processor; however, oneskilled in the art will appreciate that a hardware device may includemultiple processing elements or processors and multiple types ofprocessing elements or processors. For example, a hardware device mayinclude multiple processors or a processor and a controller. Inaddition, other processing configurations are possible, such as parallelprocessors.

The computer programs include processor-executable instructions that arestored on at least one non-transitory computer-readable medium (memory).The computer programs may also include or rely on stored data. Thecomputer programs may encompass a basic input/output system (BIOS) thatinteracts with hardware of the special purpose computer, device driversthat interact with particular devices of the special purpose computer,one or more operating systems, user applications, background services,background applications, etc. As such, the one or more processors may beconfigured to execute the processor executable instructions.

The computer programs may include: (i) descriptive text to be parsed,such as HTML (hypertext markup language) or XML (extensible markuplanguage), (ii) assembly code, (iii) object code generated from sourcecode by a compiler, (iv) source code for execution by an interpreter,(v) source code for compilation and execution by a just-in-timecompiler, etc. As examples only, source code may be written using syntaxfrom languages including C, C++, C#, Objective-C, Haskell, Go, SQL, R,Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5,Ada, ASP (active server pages), PHP, Scala, Eiffel, Smalltalk, Erlang,Ruby, Flash®, Visual Basic®, Lua, and Python®.

Further, at least one embodiment of the invention relates to thenon-transitory computer-readable storage medium including electronicallyreadable control information (processor executable instructions) storedthereon, configured in such that when the storage medium is used in acontroller of a device, at least one embodiment of the method may becarried out.

The computer readable medium or storage medium may be a built-in mediuminstalled inside a computer device main body or a removable mediumarranged so that it can be separated from the computer device main body.The term computer-readable medium, as used herein, does not encompasstransitory electrical or electromagnetic signals propagating through amedium (such as on a carrier wave); the term computer-readable medium istherefore considered tangible and non-transitory. Non-limiting examplesof the non-transitory computer-readable medium include, but are notlimited to, rewritable non-volatile memory devices (including, forexample flash memory devices, erasable programmable read-only memorydevices, or a mask read-only memory devices); volatile memory devices(including, for example static random access memory devices or a dynamicrandom access memory devices); magnetic storage media (including, forexample an analog or digital magnetic tape or a hard disk drive); andoptical storage media (including, for example a CD, a DVD, or a Blu-rayDisc). Examples of the media with a built-in rewritable non-volatilememory, include but are not limited to memory cards; and media with abuilt-in ROM, including but not limited to ROM cassettes; etc.Furthermore, various information regarding stored images, for example,property information, may be stored in any other form, or it may beprovided in other ways.

The term code, as used above, may include software, firmware, and/ormicrocode, and may refer to programs, routines, functions, classes, datastructures, and/or objects. Shared processor hardware encompasses asingle microprocessor that executes some or all code from multiplemodules. Group processor hardware encompasses a microprocessor that, incombination with additional microprocessors, executes some or all codefrom one or more modules. References to multiple microprocessorsencompass multiple microprocessors on discrete dies, multiplemicroprocessors on a single die, multiple cores of a singlemicroprocessor, multiple threads of a single microprocessor, or acombination of the above.

Shared memory hardware encompasses a single memory device that storessome or all code from multiple modules. Group memory hardwareencompasses a memory device that, in combination with other memorydevices, stores some or all code from one or more modules.

The term memory hardware is a subset of the term computer-readablemedium. The term computer-readable medium, as used herein, does notencompass transitory electrical or electromagnetic signals propagatingthrough a medium (such as on a carrier wave); the term computer-readablemedium is therefore considered tangible and non-transitory. Non-limitingexamples of the non-transitory computer-readable medium include, but arenot limited to, rewritable non-volatile memory devices (including, forexample flash memory devices, erasable programmable read-only memorydevices, or a mask read-only memory devices); volatile memory devices(including, for example static random access memory devices or a dynamicrandom access memory devices); magnetic storage media (including, forexample an analog or digital magnetic tape or a hard disk drive); andoptical storage media (including, for example a CD, a DVD, or a Blu-rayDisc). Examples of the media with a built-in rewritable non-volatilememory, include but are not limited to memory cards; and media with abuilt-in ROM, including but not limited to ROM cassettes; etc.Furthermore, various information regarding stored images, for example,property information, may be stored in any other form, or it may beprovided in other ways.

The apparatuses and methods described in this application may bepartially or fully implemented by a special purpose computer created byconfiguring a general purpose computer to execute one or more particularfunctions embodied in computer programs. The functional blocks andflowchart elements described above serve as software specifications,which can be translated into the computer programs by the routine workof a skilled technician or programmer.

Although described with reference to specific examples and drawings,modifications, additions and substitutions of example embodiments may bevariously made according to the description by those of ordinary skillin the art. For example, the described techniques may be performed in anorder different with that of the methods described, and/or componentssuch as the described system, architecture, devices, circuit, and thelike, may be connected or combined to be different from theabove-described methods, or results may be appropriately achieved byother components or equivalents.

According to at least one embodiment, a computer-implemented method forthe provision of a two-dimensional visualization image having aplurality of visualization pixels for the visualization of athree-dimensional object represented by volume data is provided for auser. The method comprises:

providing context data containing natural language and/or text inrespect of the visualization of the three-dimensional object;

determining a mapping indication based on the context data, wherein themapping indication has information about a mapping of the volume dataonto the visualization image that is suitable based on the context data;

determining a mapping rule based on the mapping indication;

mapping the volume data onto the visualization pixels using the mappingrule for the creation of the visualization image; and

providing the visualization image.

It is an idea of at least one embodiment of the present invention,during the creation of visualization images, in particular by way of avolume rendering method, to evaluate available context data in respectof the visualization, with this context data containing information,which is in the form of natural language. The context data can reflectthe problem underlying the visualization. The visualization can bebetter adjusted to the underlying problem by incorporating the contextdata in the visualization pipeline. Since, according to one aspect, thecontext data can comprise, in particular, speech inputs of the user, apossibility is thus opened up for intuitively influencing thevisualization, and, more precisely, without it being necessary for theuser to have deeper knowledge about the process of volume rendering assuch.

In other words, at least one embodiment of a computer-implemented methodis provided for the provision of a two-dimensional visualization imagehaving a plurality of visualization pixels for the visualization of athree-dimensional object represented by volume data for a user. Contextinformation for the visualization is obtained in the process by theevaluation of natural language and is taken into account in thevisualization. The natural language can be in the form of electronicdocuments, which are or can be assigned to the visualization process. Inaddition, the natural language can be in the form of a speech input of auser relating to the visualization, which speech input can be madebefore, during or after the visualization.

The user can be, in particular, the recipient of the visualization andtherewith that person for who the visualization was created. The usercan be, in particular, a doctor or a patient.

The volume data can contain a plurality of voxels. A voxel (“volumepixel” or three-dimensional pixel) is a volume element, which representsa value on a regular grid in the three-dimensional space. Voxels areanalogous to pixels, which represent two-dimensional image data. As withpixels, the voxels themselves typically do not contain their position inthe space (their coordinates), instead their coordinates are derivedbased upon their positions relative to other voxels (their positions inthe data structure, therefore, which forms a single volume image). Thevalue of a voxel can represent different physical properties of thethree-dimensional object, such as a local density. In computedtomography scans (CT scans), the values are expressed, for example, inHounsfield units, which represent the opacity of a mapped material inrespect of X-rays. The volume data thereby describes a three-dimensionalobject in an object volume. In particular, the volume data can disclosea (in particular inhomogeneous) density of the three-dimensional objectin the object volume.

The volume data can be, in particular, medical image data of a patient.The volume data can be provided, in particular, by a medical imagingmethod. The imaging methods can be based, for example, on radioscopy,computed tomography (CT), magnetic resonance tomography (MR), ultrasoundand/or positron emission tomography (PET). The three-dimensional objectcan accordingly be a body or body part of a patient. Thethree-dimensional object can comprise one or more organ(s) of thepatient.

The volume data can be mapped using an image synthesis algorithm. Theimage synthesis algorithm can be understood, in particular, as acomputer program product, which is designed for mapping the volume dataonto a two-dimensional projection area or for volume rendering of thethree-dimensional body. The projection area is given by thevisualization image. The image synthesis algorithm can have programcomponents in the form of one or more instruction(s) for a processor forcalculation of the visualization image. The visualization image iscomposed of a plurality of visualization pixels. The resolution of thevisualization image in respect of the visualization pixels can be, inparticular, spatially constant or homogeneous or spatially uniform.Other terms for image synthesis algorithms are, for example, “renderer”,“render algorithm” or “volume renderer”. The image synthesis algorithmcan be provided, for example, in that it is held available in a storagefacility or is loaded into a main memory of a suitable data processingfacility or is provided generally for application.

The image synthesis algorithm can implement different methods for thevisualization of a volume data set individually or in combination. Forexample, the image synthesis algorithm can have a ray casting moduleand/or a path tracing module.

Providing the volume data can comprise, for example, holding availableand/or retrieving the volume data in or from a storage facility and/orloading the volume data, for example into a main memory of a suitabledata processing facility, or generally making it available for anapplication or a use. A plurality of different volume data can exist orbe provided for a patient in this case.

Providing the context data can comprise, for example, holding availableand/or retrieving the context data in or from a storage facility and/orloading the volume data, for example into a main memory of a suitabledata processing facility, or generally making it available for anapplication or use. The natural language can comprise, for example, textor spoken word.

Context data in respect of the visualization can refer to informationrelevant to the visualization. It can indicate, for example, whichperspectives, transfer functions, or which partial objects of the objectto be visualized are particularly relevant to the visualization. Forexample, the context data can literally disclose which partial object,or in the case of medical volume data, which organ, is particularlyrelevant to the visualization. If a medical report (as a form of contextdata) contains, for example, statements on the liver of the patient, itis possible to conclude that this organ should primarily be displayed inthe visualization. Information of this kind is referred to as a mappingindication.

The mapping indication is converted into a mapping rule, which can beunderstood, in particular, as an instruction as to how the mappingindication can be suitably implemented in the context of avisualization. Returning to the above example, for optimum visualizationof the liver, an optimum perspective, optimum scene lighting, an optimumcoloring can be selected for the mapping of the volume data. Inaddition, fewer relevant regions of the volume data can be omitted orcut away or represented transparently in the visualization.

According to at least one embodiment, the volume data is mapped by avolume rendering method, in particular using an ray casting methodand/or path tracing method.

Particularly realistic visualizations can be generated by methods ofthis kind, and this increases the benefit of the method. The methods arecomplex in application but the inventive consideration of the contextdata means the methods can be easily operated and automatically providean optimum visualization.

According to at least one embodiment, determining the mapping rulecomprises one or more of the following step(s):

establishing a suitable perspective;

establishing a suitable scene illumination;

establishing a suitable transfer function;

establishing a suitable transparency of one or more region(s) of thevolume data;

establishing a suitable cutting plane through the volume data;

establishing a suitable region of interest inside the volume data;and/or

establishing one or more suitable item(s) of additional information foroverlaying in the visualization image.

The visualization can consequently be optimally adjusted to the contextdata.

Transfer functions often comprise representation parameters.Representation parameters are often very complex. They assign to eachgray value in the three-dimensional volume a particular color,transparency, contrast, illumination, definition and the like. Ingeneral terms, the representation parameters influence the type ofrepresentation of objects of the corresponding object type in thevisualization image output to the user.

According to a further implementation of at least one embodiment, themethod can also comprise:

determining, based on the mapping indication, a sequence of individualvisualization images to be created,

determining a mapping rule in each case for the individual visualizationimages,

providing the individual visualization images based on the mappingrules,

providing a visualization stream (or a video) based on the individualvisualization images.

The user can be automatically offered tracking shots through the volumedata to be visualized by the provision of the visualization stream, andthis can simplify the perception of specific medical issues, which canbe indicated by the context data.

According to one development of at least one embodiment, the method canalso comprise:

determining progression parameters for the provision of the individualvisualization images based on the mapping indication, wherein theprogression parameters are selected from a progression of a perspective,a progression of an enlargement factor, a progression of an arrangementposition of a clipping plane and/or a progression of a transparencysetting.

In particular, a rotation or the selective observation of organs or oforgan systems can thus take place.

According to one implementation of at least one embodiment, the volumedata comprises a plurality of different volume data sets of the patientand the method also comprises the following steps:

selection or one or more suitable volume data set(s) from the pluralityof provided volume data sets based on the mapping indication, whereinthe mapping indication and/or the mapping rule is determined in such away that the visualization image is generated based upon the selectedvolume data sets.

In other words, volume data sets for the visualization are purposefullyselected for the respective problem based on the available volume dataand the context information, and this improves the convenience and thetime expenditure for the user.

According to one implementation of at least one embodiment, at least twodifferent volume data sets are selected in the step of selection of oneor more suitable volume data set(s), with the mapping indication and/orthe mapping rule being determined in such a way that the visualizationimage is generated based upon the at least two different volume datasets.

Different volume data sets can thus be combined in the creation of avisualization image. Complementary views can be shown in a visualizationimage thereby.

According to one implementation of at least one embodiment, the at leasttwo different volume data sets comprise a first volume data set and asecond volume data set, wherein the first volume data set was generatedby a computed tomography scan of an acquisition region (of the patient)with a first radiation spectrum (of a first X-ray energy) and the secondvolume data set was generated by a computed tomography scan of theacquisition region with a second radiation spectrum (of a second X-rayenergy different from the first) different from the first.

Consequently, two scans, which were obtained with different radiationspectra or at different energy levels, are overlaid and evaluated. Thisallows improved images to be calculated since, for example, artifacts,which are caused by beam hardening, can be compensated.

According to at least one embodiment, the object has a plurality ofindividual objects, which are mapped in the volume data set.Furthermore, determining the mapping rule comprises the selection of oneor more of the individual object(s) for representation in thevisualization image. In particular, a separate transfer function,optimized for the respective individual object, can be used for eachselected individual object.

According to one further embodiment, determining the mapping rule alsocomprises adjusting the mapping rule based on the selected one or moreindividual object(s).

Consequently, a mapping rule suitable for the selected individualobjects can be automatically generated, and this eases the workload ofthe user in having to look for suitable parameters for visualizationthemselves.

The visualization can thereby be purposefully directed to the individualobjects indicated by the context data. In particular, the individualobjects can be different organs or anatomies of a patient.

According to one embodiment, the method also comprises the step ofdetermining object information from the context data, with the step ofselecting taking place based upon the object information.

The object information can indicate, for example, which of theindividual objects are relevant to a visualization. If the context datacontains, for example, one or more voice command(s) of the user, whichare directed to an individual object (for example the liver of thepatient) this can be taken into account accordingly in thevisualization. The same applies if the context data comprises, forexample, one or more medical report(s), in which one or more individualobject(s) are mentioned.

As a result, the object which is relevant can be automatically selectedfrom the context data for the visualization and the visualizationparameters can then automatically be adjusted without the user having toact.

According to one embodiment, the method also has the step of recognizingor identifying one or more individual object(s) in the volume data. Inparticular, a segmentation method and in particular a multi-organsegmentation can be used (or be applied to the volume data). The step ofselecting then takes place in particular based upon the one or moreindividual object(s) recognized in this way.

In a two-stage selection process, it is thereby possible to first of alldetermine which objects (organs) are even represented in the volumedata, whereupon it is then possible to determine by evaluation of thecontext data which of the represented objects are especially relevant toa visualization.

According to one embodiment, the method also comprises the followingstep:

acquiring acquisition information from the volume data and/or thecontext data, wherein the acquisition information contains informationabout how the volume data was generated; wherein

the mapping indication and/or the mapping rule is also determined basedupon the acquisition information.

The acquisition information can comprise, for example, information aboutwith which imaging modality the volume data was generated. By way ofexample, the acquisition information can indicate, for example, that thevolume data was generated with a magnetic resonance device, a computedtomography device, an (4D-)ultrasound device, etc. Furthermore, theacquisition information can comprise information about with whichimaging protocols the volume data was generated, for example which MRprotocol was used, therefore, whether and which contrast medium wasadministered, whether, for example, an angiography run was carried out,etc.

Taking into account the acquisition information means the mappingindication and/or the mapping rule can be adjusted better and the volumedata can be adjusted and the quality of the visualization image can beincreased. The acquisition information can be obtained directly from thevolume data (because, for example, a particular form of volume datapoints to a particular imaging modality or this is stored in metadatarelating to the volume data) or can be contained in the context data(for example, because a medical report indicates that a particularvolume data set was generated with a CT device):

According to one embodiment, the one or more of the individual object(s)is also selected based upon the acquisition information.

Consequently, it is possible to take into account the fact thatparticular scan modalities and/or scan modes are especially well suitedto the representation of certain individual objects and others, in turn,less so. If the acquisition information comprises, for example,information to the extent that an angiography run was carried out, theselection of the vascular tree of the patient as the individual objectto be represented is with some certainty expedient.

According to one embodiment, selecting one or more individual object(s)comprises a selection of at least one virtual individual object, whichis not contained in the volume data, wherein the mapping indicationand/or the mapping rule is determined in such a way that the at leastone virtual individual object is represented in the visualization image.

According to some implementations, the at least one virtual individualobject can be represented in second volume data different from thevolume data. In this connection, the term virtual should be taken tomean “not contained in the volume data”. The second volume data canlikewise be volume data of the patient. The second volume data can havebeen acquired in particular with an imaging modality, which is differentfrom an imaging modality with which the volume data was acquired. Forexample, the volume data can have been acquired with an MR device whilethe second volume data was acquired with a CT device. Consequently,complementary views may advantageously be combined in one visualizationimage. According to further implementations, the second volume data canbe volume data of a second patient, which differs from the patient.Consequently, comparisons between different patients are made possible.According to further implementations, the virtual individual object canrefer to an artificial object such as an implant. The geometric and/oroptical properties can be in the form, for example, of separate objectdata.

According to some implementations, the step of providing the mappingindication and/or the mapping rule comprises registering the at leastone virtual individual object with the volume data (or the second volumedata with the volume data and/or the object data with the volume data),with the mapping indication and/or the mapping rule being determinedbased upon the registration.

According to some implementations, the method also comprises selectingthe second volume data from a plurality of volume data sets of thepatient based on the volume data and/or the context data.

Consequently, second volume data can be purposefully selected for apatient and be used for incorporation of the at least one virtualindividual object in the visualization.

According to one embodiment, the method includes receiving one or morespeech input(s) of a user, with the context data being provided basedupon the speech inputs of the user.

In other words, the visualization can consequently be voice-controlled.The user can indicate by way of a voice command, for example, whichpartial object or which organ he would willingly like to see. This isinventively automatically converted into a mapping rule. Consequently,especially simple, intuitive operation of the visualization method ispossible.

According to one embodiment, the speech input is received afterproviding the visualization image and the method also has the followingsteps:

adjusting the mapping rule based on the context data;

mapping the volume data onto the visualization pixels with the imagesynthesis algorithm using the adjusted mapping rule for the creation ofan adjusted visualization image; and

provision of the adjusted visualization image.

Consequently, continuous interactive adjustment of the visualization byway of speech input is possible.

According to one embodiment, receiving one or more speech input(s) ofthe user comprises:

receiving an audio signal containing natural language of the user,

a beginning of a speech input of the user in the audio signal,

analysis of the speech input of the user in order to identify one ormore voice command(s) of the user directed to the visualization of thevolume data, and

providing the context data based on the one or more voice command(s).

The audio signal can contain, in particular, sound information. Theaudio signal can be an analog or a digital or digitalized signal. Thedigitalized signal can be generated starting from the analog signal forexample by an analog-to-digital converter. Accordingly, the step ofreceiving can comprise a step of providing a digitalized audio signalbased on the received audio signal or a digitalization of the receivedaudio signal. Receiving the audio signal can comprise, in particular,registering the audio signal by way of a suitable sensor, for example anacoustic sensor such as a microphone.

The audio signal can comprise communication of a user, such as aninstruction to be carried out or a question. In other words, the audiosignal can comprise a speech input of the operator in natural language.Typically, speech spoken by humans is referred to as natural language.Natural speech can also have an inflection and/or intonation formodulation of the communication. In contrast to formal speech, naturallanguage can have structural and lexical ambiguities.

It can be provided firstly to recognize the beginning of such a speechinput in the audio signal. In other words, an activity recognition canbe carried out, which shows when the audio signal contains spokenspeech. From the instant at which a speech input was recognized, theaudio signal is relevant to further analysis in order to infer, forexample, the content of the speech input of the user. The recognition ofa beginning of a speech input can comprise, in particular, recognitionof a human voice in the audio signal. This can occur by way of signalanalysis by, for example, values such as frequency, amplitude,modulations, etc. being recognized in the audio signal, which arecharacteristic of the human voice.

The speech input contained in the audio signal is continuously analyzedfrom the recognized beginning of the speech input. Accordingly, a resultof speech analysis, which contains one or more recognized voicecommand(s), is provided. A method for processing natural language can beused for this purpose. In particular, a computer linguistics algorithmcan be applied to the audio signal or the speech input for this purpose.One possibility of processing the speech input formulated in naturallanguage for the provision of the result of speech analysis isconverting the speech input formulated in natural language into text(structured speech, therefore), by way of a text-to-speech (software)module. A further analysis for the provision of the result of speechanalysis, for example by way of latent semantic indexing (LSI forshort), can then ascribe meaning to the text. Speech analysis and, inparticular, an understanding of the natural language contained in thespeech input (natural language understanding, NLU) can then occur.

Providing the (adjusted) visualization image can comprise an output ofthe (adjusted) visualization image to the user via a user interface, atransfer of the (adjusted) visualization image to a storage facility(for instance a PACS system) and/or embedding of the adjustedvisualization image in a medical report.

According to one implementation, the method comprises receiving one ormore gesture command(s) of a user, with the context data being providedbased upon the gesture commands of the user.

Gesture commands can comprise, for example, the user pointing to orlooking at a visualization area of the visualization image. Gesturecommands can be detected, for example, by way of a suitable cameraapparatus and be registered with the visualization image. Gesture andlook detection algorithms that are known per se can be used in thiscase.

According to one implementation of at least one embodiment, the methodcomprises receiving one or more combined gesture command(s) and speechinput(s) of a user, with the context data being provided based upon thegesture commands of the user.

Consequently, the user can influence the visualization by way ofcombined gestures and voice control. For example, he can point to avisualization area of the visualization image and also give aninstruction (for example “make this area brighter”).

According to one embodiment, the context data has information about oneor more preference(s) of the user in respect of the visualization of thevolume data.

A visualization can be proactively adjusted to the preferences of theuser by taking account of the user preferences. The user has to makeeven fewer adjustments, and this makes the method easier to operate and,in particular in clinical practice, more useful.

According to one embodiment, the context data comprises one or moreelectronic document(s) containing natural language and/or text relatingto the object to be visualized. The documents can comprise, inparticular, one or more medical report(s) and/or medical guideline(s).

A presumably expedient visualization can be determined by the evaluationof clinical reports and/or medical guidelines and can be automaticallyoffered to the user.

According to some implementations of embodiments, the electronicdocuments can also comprise, for example, doctor's letters, metadata ofthe volume data (for example DICOM attributes), one or more clinicalontologies such as RadLex or SNOMED, and/or comparative medical studies.

According to some implementations of embodiments, the method alsocomprises:

providing an evaluation algorithm for application to the volume data,with the evaluation algorithm being designed to provide an analyticalresult on application to volume data;

generating an analytical result by applying the evaluation algorithm tothe volume data;

providing the context data also based upon the analytical result.

An analytical result can be directly taken into account in thevisualization due to this step. If the evaluation algorithm is designed,for example, to detect lesions in the lungs, the mapping indicationand/or the mapping rule can be determined in such a way that the lesionsare emphasized accordingly. An evaluation algorithm can generally bedesigned to evaluate volume data or the image information containedtherein for a specific, in particular medical problem. An evaluationalgorithm can thus be designed, for example, to recognize and classifyobjects, to measure geometries, and/or to mark objects.

According to one implementation of at least one embodiment, providingthe evaluation algorithm also comprises:

providing a plurality of different evaluation algorithms, which is ineach case designed to provide a specific analytical result onapplication to volume data;

selecting the evaluation algorithm from the different evaluationalgorithms based on the volume data and/or the context data.

Consequently, a suitable evaluation algorithm can be automaticallyselected based upon the clinical context or even a verbal user input,and this further improves the convenience.

According to one embodiment, one or more of the electronic document(s)can be retrieved from a database. In the medical application, forexample an electronic patient identifier, such as an ID or a name of thepatient, can be extracted from the volume data for this purpose, withwhich identifier the database can then be searched for relevantdocuments. The electronic documents can be part of an electronic file ofthe patient. The database can be designed, for example, to storeelectronic patient files of a plurality of patients. The database canbe, for example, part of a hospital information system.

According to one embodiment, the volume data is not provided in advancebut retrieved from a suitable database based upon the context data.Based upon the context data, a suitable series, suitable volume data canbe retrieved thereby, with which, for example, the mapping indicationmay be implemented especially well.

According to one embodiment, the visualization image has a series of aplurality of individual images, which visualize the volume data or aseries of volume data in a time-resolved manner. In other words, thevisualization image can contain a video.

According to one embodiment, the mapping indication and/or the mappingrule is determined with a configuration algorithm. The configurationalgorithm can comprise, in particular, a computer linguistics algorithm.

The configuration algorithm can be understood, in particular, as acomputer program product, which is designed for determination of themapping indication or the mapping rule. The configuration algorithm canhave program components in the form of one or more instruction(s) for aprocessor for determination of the mapping indication or the mappingrule. The configuration algorithm can be provided, for example, in thatit is held available in a storage facility or is loaded into a mainmemory of a suitable data processing facility or is generally providedfor application or use.

The computer linguistics algorithm can be designed, in particular, foralgorithmic processing of natural language in the form of text or voicedata with the aid of a computer. The computer linguistics algorithm canbe designed, in particular, to recognize or to identify in the contextdata words relevant to the visualization and their context, and thustransfer them into a mapping indication or to generate a mappingindication.

According to one embodiment, the configuration algorithm, and inparticular the computer linguistics algorithm, has one (or more) trainedfunction(s).

A trained function generally maps input data onto output data. Here theoutput data can still depend, in particular, on one or more parameter(s)of the trained function. The one or more parameter(s) of the trainedfunction can be determined and/or adjusted by training. Determiningand/or adjusting of the one parameter or the plurality of parameters ofthe trained function can be based, in particular, on a pair of traininginput data and associated training output data, with the trainedfunction being applied to training input data for the generation oftraining mapping data. In particular, determining and/or adjusting canbe based on a comparison of the training mapping data and the trainingoutput data. In general, a trainable function, in other words a functionwith parameters that have not yet been adjusted, is also referred to asa trained function.

Other terms for trained function are trained mapping rule, mapping rulewith trained parameters, function with trained parameters, algorithmbased on artificial intelligence, machine learning algorithm. Oneexample of a trained function is an artificial neural network. Insteadof the term “neural network” the term “neural net” can also be used.Basically, a neural network is constructed like a biological neuralnetwork such as a human brain. In particular, an artificial neuralnetwork comprises an input layer and an output layer. It can alsocomprise a plurality of layers between input and output layers. Eachlayer comprises at least one, preferably a plurality of nodes. Each nodecan be taken to mean a biological processing unit, for example a neuron.In other words, each neuron corresponds to an operation, which isapplied to input data. Nodes of a layer can be connected by edges orconnections to nodes of other layers, in particular by directed edges orconnections. These edges or connections define the flow of data betweenthe nodes of the network. The edges or connections are associated with aparameter, which is often referred to as a “weight” or “edge weight”.These parameters can regulate the importance of the output of a firstnode for the input of a second node, with the first node and the secondnode being connected by an edge.

In particular, a neural network can be trained. In particular, trainingof a neural network is carried out based upon the training input dataand associated training output data according to a “supervised” learningtechnique, with the known training input data being input into theneural network and the output data generated by the network beingcompared with the associated training output data. The artificial neuralnetwork learns and adjusts the edge weights for the individual nodesindependently as long as the output data of the last network layer doesnot sufficiently correspond to the training output data.

In particular, a trained function can also be deep neural network ordeep artificial neural network.

According to some implementations of at least one embodiment,determining the mapping rule comprises providing a trained function,which is adjusted in such a way that, based on volume data and contextdata, it provides a mapping rule suitable for the context data, andproviding the mapping rule by applying the trained function to thecontext data and the volume data.

According to some implementations of at least one embodiment, thetrained function has a neural network and in particular a convolutionalneural network.

In particular, the convolutional neural network can be designed as adeep convolutional neural network. The neural network has one orconvolutional layer(s) and one or more deconvolutional layer(s). Inparticular, the neural network can comprise a pooling layer. A neuralnetwork can be especially efficiently used for the derivation of amapping rule through the use of convolutional layers and/ordeconvolutional layers since only a few edge weights (namely the edgeweights corresponding to the values of the convolutional core) have tobe determined despite many connections between node layers. With anidentical number of training data items, the accuracy of the neuralnetwork can also be improved thereby.

According to one example embodiment, the trained function has a U-netarchitecture.

A U-net architecture is based on convolutional neural networks, withpooling layers being replaced by “upsampling” layers. In addition, jumplinks are provided, which enable context information to be input, forexample from convolutional layers, directly into deconvolutional layers.Faster processing can thus be achieved through the use of a U-netarchitecture and less training data is required.

According to a further embodiment, a computer-implemented method forproviding a trained function for determining a mapping rule isdisclosed. The method comprises:

providing training input data, wherein the training input data hascontext data containing natural language and/or text in respect of thevisualization of a three-dimensional object and volume data representingthe three-dimensional object;

providing training output data, wherein the training output data has amapping rule for the generation of a suitable two-dimensionalvisualization image by mapping the volume data;

generating a training mapping rule by applying the trained function tothe training input data;

comparing the training mapping rule with the training output data;

adjusting the trained function based upon the comparison.

Training input and training output data can be provided, for example, inthat suitable mapping rules for particular volume data and context dataare determined by a visualization expert. Consequently, the trainedfunction is rendered capable of reproducing suitable mapping rules forunknown volume data/context data and thus saves a user who is lessexperienced in respect of visualization the time-consuming task offinding suitable visualization parameters.

The use of a trained function should be understood as being merelyoptional. Alternatively, other solutions can also be pursued, such as asolution based on strictly predefined rules.

According to a further embodiment, an apparatus is disclosed for theprovision of a two-dimensional visualization image having a plurality ofvisualization pixels for visualization of a three-dimensional objectrepresented by volume data for a user, which apparatus comprises:

an interface, which is designed to receive the volume data and contextdata containing natural language in respect of the visualization of thethree-dimensional object;

a configuration module, which is designed to:

-   -   determine a mapping indication based on the context data (and        optionally on the volume data), wherein the mapping indication        comprises information about a mapping of the volume data, which        is suitable based on the context data, onto the visualization        image; and    -   determine a mapping rule based on the mapping indication; and

a visualization module, which is designed to:

-   -   map the volume data onto the visualization pixels using the        mapping rule for the creation of the visualization image; and    -   provide the visualization image.

The configuration module and/or the visualization module of anembodiment can be part of an computing facility. The configurationmodule can be designed to carry out the configuration algorithm. Thevisualization module can be designed to carry out the image synthesisalgorithm.

The computing facility can be designed as a central or decentralcomputing facility. The computing facility can have one or moreprocessor(s). The processors can be designed as a central processingunit (CPU for short) and/or as graphics processing unit (GPU for short).The computing facility can be designed as what is known as asystem-on-a-chip (SoP for short), which controls all functions of adevice. Alternatively, the computing facility can be implemented as alocal or Cloud-based processing server.

The interface of an embodiment can be designed generally for theexchange of data between the computing facility and further components.The interface can be implemented in the form of one or more individualdata interface(s), which can have a hardware and/or software interface,for example a PCI bus, a USB interface, a Firewire interface, a ZigBeeinterface or a Bluetooth interface. The interface can also have aninterface of a communications network, wherein the communicationsnetwork can have a Local Area Network (LAN), for example an Intranet ora Wide Area Network (WAN). The one or more data interface(s) canaccordingly have a LAN interface or a Wireless LAN interface (WLAN orWi-Fi).

The advantages of the proposed apparatus substantially match theadvantages of the proposed method. Features, advantages or alternativeembodiments can likewise be transferred to the other claimed subjectmatters, and vice versa.

According to one embodiment, the apparatus also has a storage facility,which is designed to store context data and provide it via theinterface.

The storage facility can be, in particular, part of what is known as aPicture Archiving and Communication system (PACS). In addition oralternatively, the storage facility can be part of a medical informationsystem, such as a hospital or laboratory information system.

The invention relates in a further embodiment to a computer programproduct, which comprises a program and can be loaded directly into astorage device of a programmable computing facility and has programcode/segments, for example libraries and auxiliary functions, in orderto carry out an embodiment of a method for the visualization of athree-dimensional object in particular according to the embodiment whenthe computer program product is run.

Furthermore, the invention relates in a further embodiment to a computerprogram product, which comprises a program and direct can be loadeddirectly into a storage device of a programmable computing facility andhas program code/segments, for example libraries and auxiliaryfunctions, in order to carry out a method for the provision of a trainedfunction in particular according to the embodiment when the computerprogram product is run.

The computer program products of an embodiment can comprise softwarewith a source code, which still has to be compiled and linked or whichonly has to be interpreted, or an executable software code, which forexecution only has to be loaded into the processing unit. As a result ofthe computer program products the method can be carried out quickly,repeatedly in an identical manner and robustly. The computer programproducts are configured such that they can carry out the inventivemethod steps by way of the computing facility. The computing facilityhas to have in each case the requirements such as an appropriate mainmemory, an appropriate processor, an appropriate graphics card or anappropriate logic unit, so the respective method steps of an embodimentcan be efficiently carried out.

The computer program products are stored, for example in an embodiment,on a computer-readable storage medium or saved on a network or serverfrom where they can be loaded into the processor of the respectivecomputing facility, which is directly connected to the computingfacility or can be designed as part of the computing facility.Furthermore, control information of the computer program products can bestored on a computer-readable storage medium. The control information ofthe computer-readable storage medium can be configured in such a waythat it carries out an inventive method when the data carrier is used inan computing facility. Examples of computer-readable storage medium area DVD, a magnetic tape or a USB stick on which electronically readablecontrol information, in particular software, is stored. When thiscontrol information is read from the data carrier and stored in ancomputing facility, all inventive embodiments of the previouslydescribed method can be carried out. The invention can thus also startfrom the computer-readable medium and/or the computer-readable storagemedium. The advantages of the proposed computer program product or theassociated computer-readable media substantially match the advantages ofan embodiment of the proposed method.

FIG. 1 shows a system 1 for the provision of a visualization image for auser N according to one embodiment of the invention in a schematicrepresentation.

The system 1 has a storage unit 60 and a database 70, a visualizationmodule 23 and a configuration module 22, which have a data connection toeach other.

The storage unit 30 is designed to store volume data. The volume datais, in particular, image data, which was acquired from patients usingmedical imaging modalities. The volume data is linked to one patient ineach case thereby.

The database 70 is designed to store context data KD for a visualizationof the volume data. The context data KD can comprise, in particular,patient context data. This patient context data is linked to one patientin each case and can consequently be assigned to the volume data. Thepatient context data can comprise, for example, medical findings,reports, doctor's letters and the like. The patient context data can beprovided, for example, in the form of an electronic medical record.Furthermore, the context data KD can include user context data, which isspecific to a user N of the system 1. For example, the user context datacan contain user preferences in respect of the visualization of volumedata. In addition, the context data can contain higher-order clinicaldata, which can comprise, for example, one or more medical guideline(s).

The storage unit 60 and/or the database 70 can be designed asdistributed storage facilities and comprise a plurality of individualstorage devices.

An additional source (apart from the database 70) for context data KDcan be speech inputs of the user N. Generally, the context data KD haselements, which include natural language and/or text (for example in theform of the speech inputs or as text for example in the patient contextdata and/or clinical guidelines).

The configuration module 22 is designed to determine a suitablevisualization of volume data VD for a patient based on an analysis ofthe context data KD and to derive a mapping rule for the visualizationmodule 23 therefrom. For this purpose, the configuration module 22 isdesigned, in particular, to evaluate the elements natural languageand/or text in the context data KD. The configuration module 22 canimplement an appropriately configured algorithm for this purpose. Thisalgorithm can comprise a computer linguistics algorithm for processingnatural language and/or text.

The visualization module 23 is designed to map volume data VD onto thevisualization image based upon a mapping rule and to thus generate avisualization. The visualization module 23 can implement a visualizationalgorithm for this purpose.

If a user N wishes to see a medical record in order, for example, tomake a medical diagnosis, the user N can be inventively assisted in thisas follows. Firstly, volume data VD is provided in step S10.

In a step S20, the configuration module 22 automatically, or in responseto a user activation, loads or collects relevant context data KD. Theconfiguration module 22 can, for example, receive a speech input fromthe user N or query the database 70 for relevant context data for thispurpose.

Based upon the context data, the configuration module 10 is designed todetermine a mapping indication, which shows which type of mapping of thevolume data is probably expedient or desired in the present case for theuser N (step S30). The mapping indication can show which perspective,which organ, which volume data or generally which mapping parameterswill probably be required. In one example, it can emerge from thecontext data, for example, that a time-resolved video sequence of theheartbeat should be rendered. This can be the case, for example, whenthe context data points to a pending cardiological examination of thepatient.

In an alternative embodiment, the volume data is already predefined (forinstance by a user selection) and taken into account on determination ofthe mapping indication.

Based upon the mapping indication, the configuration module 10 thengenerates a mapping rule for the visualization module 20 in step S40 andinputs it therein. Based upon this mapping rule, the visualizationmodule 20 then generates a visualization in step S50 and provides theuser N with it in step S60.

In a step S70, the user N then has the option of providing theconfiguration module 10 with feedback via the visualization image, whichfeedback the configuration module 22 can then use for adjustment of themapping indication and the mapping rule AV. Consequently, the user N isgiven the option of easily influencing the visualization module 23 byspeech input without having to intervene in the parameter controlthereof themselves.

FIG. 2 represents a system 1 for the visualization of athree-dimensional object according to a further embodiment. The system 1has an computing facility 20, an interface 30, a detection apparatus 40,a rendering apparatus 50, a storage unit 60 and a database 70. Thecomputing facility 20 is basically designed for calculation of avisualization image VB of a three-dimensional object based upon volumedata VD describing the three-dimensional object. The computing facility20 can be provided with the volume data VD from the storage unit 60 viathe interface 30.

The storage unit 60 can be designed as a central or decentral database.The storage unit 60 can be, in particular, part of a server system. Thestorage unit 60 can be, in particular, part of what is known as aPicture Archiving and Communication system (PACS). The volume data VDcan have been generated by a medical imaging method. For example, thevolume data VD can have been generated by radioscopy, computedtomography (CT), magnetic resonance tomography (MR), ultrasound and/orpositron emission tomography (PET). The volume data VD can be formatted,for example, in the DICOM format. DICOM stands for Digital Imaging andCommunications in Medicine and denotes an open standard for the storageand exchange of information in medical image data management. The volumedata VD has a three-dimensional data set comprising a plurality ofvolume pixels, what are known as voxels. The voxel values can beobtained, for example, by the medical imaging method and conventionallyconstitute a measure of the local density of the three-dimensionalobject at the location of the voxels. Providing the volume data VD cancomprise, for example, loading the volume data VD into a main memory(not shown) of the computing facility 20.

The database 70 can be, in particular, part of a medical informationsystem, such as a hospital information system, a PACS system, alaboratory information system and/or further medical informationsystems. The database 70 can also be designed as what is known as aCloud storage device. The database 70 can be designed to store patientdata PD. Patient data PD can be, for example, examination results, whichare not based on medical imaging. In addition, patient data PD cancomprise text data sets, such as structured and unstructured medicalreports. Alternatively or in addition, the patient data PD can alsocomprise an electronic medical record (or EMR for short) of the patient.In alternative embodiments, the database 70 can also be integrated inthe storage facility 50. Furthermore, patient data PD can beincorporated in the volume data VD for example as metadata.

The computing facility 20 is designed to take into account context dataKD in the calculation of a visualization or mapping of thethree-dimensional object. For this purpose, the computing facility 20can be provided with an audio signal AS via the interface 30, whichaudio signal AD can comprise a speech input of the user N. Furthermore,the computing facility 20 can be provided with acquisition informationEI via the interface 30, which acquisition information EI containsinformation about where the user N is currently looking or whether theuser N is currently gesturing. The audio signal AS and the acquisitioninformation EI are provided by the detection apparatus 40. The detectionapparatus 40 can accordingly be configured to detect a viewing directionof a user N. For this purpose, it can have optical detection device(s)such as cameras, which detect the eye area and/or the pupil positionand/or the K head position of the user N. The acquisition information EIcan then include, for example, image data of an eye area, a pupilposition and/or a head position. The detection apparatus 40 can also beconfigured to detect a gesture of a user N. For this purpose, it canhave optical detection device(s) such as cameras, which detect agesture, for instance with a hand, of the user N. Furthermore, thedetection apparatus 40 can have an acoustic input apparatus. Theacoustic input apparatus serves to record or acquire an audio signal AS,to record spoken sounds, therefore, which are generated by the user N ofthe system 1. The acoustic input apparatus can be implemented, forexample, as a microphone. The acoustic input apparatus can bestationarily arranged, for example, in an operating space.Alternatively, the acoustic input apparatus can also be implemented tobe portable, for example as a microphone of a headset, which can becarried along by the N. Furthermore, the computing facility 20 canrequest patient data PD from the database 70 and this can be searchedfor context data KD. Text-mining algorithms for example can be used forthis purpose.

Once the visualization image VB has been calculated it should bedisplayed to the user N. For this purpose, the rendering apparatus 50 isprovided with the visualization image VB via the interface 30. Therendering apparatus 50 can have one or more screen and/or projectionfacility/ies, which are designed for the rendering of the visualizationimage VB for the user N. In particular, the rendering apparatus 50 canbe implemented as an image representation system of augmented realityglasses. Alternatively, the rendering apparatus 50 can have a screen ofa PC, laptop, tablet or smartphone.

The interface 30 can have one or more individual data interface(s),which guarantee the exchange of data between the components 20, 40, 50,60, 70 of the system 1. The one or more data interface(s) can have ahardware and/or software interface, for example a PCI bus, a USBinterface, a Firewire interface, a ZigBee interface or a Bluetoothinterface. The one or more data interface(s) can have an interface of acommunications network, wherein the communications network can have aLocal Area Network (LAN), for example an Intranet or a Wide Area Network(WAN). The one or more data interface(s) can accordingly have a LANinterface or a Wireless LAN interface (WLAN or Wi-Fi).

The computing facility 20 can comprise, for example, a processor, forexample in the form of a CPU or the like. The computing facility 20 canbe designed as a central control unit, for example as a control unitwith one or more processor(s). According to further implementations,functionalities and components of the computing facility 20 can bedistributed in a decentralized manner over several arithmetic units orcontrollers of the system 1.

For calculation of the visualization image VB based on the input datathe computing facility 20 can have different elements 21, 22, and 23.The element 21 can be understood as a context data provision module.Element 21 can implement or control an algorithm which is designed todetermine a viewing direction of the user N or recognize a gesturecontrol of the user N based on the acquisition information EI. For thispurpose, the acquisition information EI is evaluated by element 21 andrelated to the visualization image VB displayed or to be displayed. Forthis purpose, it is possible to draw on, for example, a suitableregistration—for instance between the detection apparatus 40 and therendering apparatus 50. Element 21 can also implement or control analgorithm, which extracts one or more voice command(s) of the user Nfrom an audio signal AS and provides it/them as context data KD. Element21 can also implement or control an algorithm, which retrieves relevantpatient data PD from the database 70 and searches for informationrelevant to the visualization and provides it as context data KD.

The element 22 can be understood as a configuration module, whichsuitably configures the visualization based on the volume data VD andthe context data KD. In particular, the element 22 can cause analgorithm to be applied, which determines a mapping indication based onthe context data KD and the volume data, under which indication thevisualization will be considered. In other words, the mapping indicationcomprises a context, in particular a medical one, which is relevant tothe visualization. Furthermore, the algorithm can be designed todetermine a mapping rule AV based on the mapping indication, which inturn is suitable for generating a mapping of the volume data VD suitablefor the mapping indication.

The element 23 can be understood as a volume rendering engine. Theelement 23 can implement or control an image synthesis algorithm, whichis designed to map the volume data VD onto the visualization image VB orthe pixels of the visualization image VB. The pixels of thevisualization image VB will be called visualization pixels below. Forcalculation of the mapping onto the visualization pixels the imagesynthesis algorithm can have different visualization modules, which canbe selectively activated and deactivated, and in particular for eachvisualization pixel, and/or can be adjusted in terms of their computingeffort. The visualization modules can refer, for example, to differentapproaches for calculation of a visualization image VB. For example, theimage synthesis algorithm can comprise a ray casting module in whichvisualization pixels VP are calculated with the method of ray casting.Furthermore, the image synthesis algorithm can have a path tracingmodule in which visualization pixels are calculated according to themethod of path tracing. In addition, the visualization modules can referto supplementary mapping effects. These supplementary mapping effectscan comprise, in particular with ray casting methods, for exampleambient occlusion effects, shadow effects, translucence effects, colorbleeding effects, surface shading, complex camera effects and/orlighting effects due to random ambient lighting conditions. Overall, theelement 23 is thereby designed in such a way that the computing effort,and therewith the quality of the mapping, can be selectively set foreach individual visualization pixel VP.

The performed division of the computing facility 20 into elements 21-23serves solely to simplify the explanation of the mode of operation ofthe computing facility 20 and should not be understood as limiting. Theelements 21-23 or their functions can also be combined into one element.The elements 21-23 can be understood in particular also as computerprogram products or computer program segments, which when run in thecomputing facility 20 implement one or more of the method step(s)described below.

FIG. 3 represents a schematic flowchart of a method for visualization ofa three-dimensional object. The order of the method steps is limited byneither the represented sequence nor by the chosen numbering. The orderof the steps can thus optionally be interchanged and individual stepscan be omitted.

A first step S10 is directed toward providing the volume data VD. Theprovision can be implemented by a retrieval of the volume data VD fromthe storage unit 60 and/or loading of the volume data VD into thecomputing facility 20.

In a next step S20, context data KD is provided for the visualization.The context data KD can comprise:

one or more voice command(s) of the user N,

one or more item(s) of information from patient data PD,

one or more user preference(s) for the visualization,

one or more gesture(s) of the user N, and/or

an analytical result of an image data evaluation algorithm applied tothe volume data VD.

Accordingly, step S20 can have a plurality of optional sub-steps. Asub-step S21 can be directed to detecting one or more voice command(s)in the audio signal AS and the provision of the voice commands ascontext data KD. A sub-step S22 can be directed to detecting one or moregesture control command(s) in the acquisition information EI and theprovision of the gesture control commands as context data KD. A sub-stepS23 can be directed to retrieving patient data PD from the database 70(or the storage unit 60), an analysis of the patient data for theprovision of a text analysis result and the provision of the textanalysis result as context data KD. A sub-step S24 can be directed toretrieving user preferences from the database 70 (or the storage unit60) and the provision of the user preferences as context data KD. Asub-step S25 can be directed to applying a selection of an evaluationalgorithm, applying the evaluation algorithm for the generation of ananalysis result and providing the analysis result as context data KD.

Speech analysis algorithms, text analysis algorithms, image analysisevaluation algorithms and the like, which are known per se, can be usedfor the provision of the context information KD.

In a next step S30, a mapping indication is determined based upon thecontext data KD. The mapping indication indicates which visualization issuitable or expedient for the user N in view of the volume data VD andthe context data. For this purpose, a trained function can be applied tothe volume data VD and the context data KD in step S30, which functionwas designed to derive a corresponding mapping indication based on thevolume data VD and the context data KD.

In step S40, a mapping rule AV is then determined based upon the mappingindication, which rule is capable of mapping the volume data VD inaccordance with the mapping indication onto the visualization image.Different parameters for mapping the volume data VD can be suitablyadjusted for this purpose, such as: a perspective, a scene illumination,a transfer function, a transparency of one or more region(s) of thevolume data, a cutting plane through the volume data, a region ofinterest within the volume data, and/or one or more additional item(s)of information for overlaying in the visualization image.

Optionally, one or more specific object(s) can be selected in step S40for visualization (step S41). In particular, first of all objects can beidentified in the volume data VD and then the context data KD selectedaccordingly for the visualization (or not selected and thus notrepresented). Furthermore, virtual objects can also be supplemented fromother volume data or further sources and also be mapped in thevisualization.

Optionally, specific volume data sets can also be selected in step S40from the available volume data VD of a patient (step S42). Thus,depending on context data KD and mapping indication, for example volumedata sets can be selected, which represent a relevant problem from thecontext data KD/the mapping indication especially well. If the contextdata KD indicates, for example, that a vessel structure should berepresented in the visualization image, angiography data sets can besought in the volume data VD and can (optionally additionally) form thebasis of the visualization. A plurality of different volume data setscan also be explicitly combined in order to obtain a suitablevisualization (for instance dual energy data sets in CT imaging).

The trained function can in turn also be suitably designed fordetermination of the mapping rule based on the mapping indication.Alternatively, the trained function or the processing generally can alsobe designed in such a way that the mapping rule is determined directlybased upon the context data KD. Step S30 can then also be omittedaccordingly and/or be included in step S40.

In step S50, the visualization image VB is then created with the mappingrule AV. In step S60, the rendering apparatus 50 is provided with thefinished visualization image VB via the interface 30. In step S70, therendering apparatus 50 displays the visualization image VB to the user.Steps S60 and S70 are optional and can be omitted.

Step S80 represents an optional repetition step. Once the visualizationimage VB has been displayed to the user N, the processing jumps via stepS80 again to step S20 and the context data KD is acquired once again.This primarily takes account of the case where the user N, based uponthe visualization image VB, has verbalized a voice command for adjustingsame. On this basis, a new mapping rule AV is calculated in step S40 andthe visualization image VB is updated in the steps S50 and optionallyS60, and S70. The calculation of the visualization image VB can becontinuously adjusted to the instantaneous context by the repetitionstep S80.

FIG. 4 shows an example representation of a trained function TF, as canbe used in step S30 or S40 for determination of a mapping rule AV. Thetrained function TF receives the context data KD as an input and outputsthe mapping rule AV as an output, in other words control commands foractuation of the visualization module 23, therefore.

In the illustrated example embodiment, the trained function TF isdesigned as a neural network. The neural network can also be referred asan artificial neural network or a neuronal network.

The neural network 100 comprises nodes 120, . . . , 129 and edges140,141, with each edge 140,141 being a directed connection of a firstnode 120, . . . , 129 to a second node 120, . . . , 129. In general, thefirst node 120, . . . , 129 and the second node 120, . . . , 129 aredifferent nodes; it is also possible that the first node 120, . . . ,129 and the second node 120, . . . , 129 are identical. An edge 140,141of a first node 120, . . . , 129 to a second node 120, . . . , 129 canalso be referred to as an incoming edge for the second node and as anoutgoing edge for the first node 120, . . . , 129.

The neural network 100 responds to input values x⁽¹⁾ ₁, x⁽¹⁾ ₂, x⁽¹⁾ ₃relating to a large number of input nodes 120, 121, 122 of the inputlayer 110. The input values x⁽¹⁾ ₁, x⁽¹⁾ ₂, x⁽¹⁾ ₃ are applied in orderto generate one or a large number of output(s) x⁽³⁾ ₁, x⁽³⁾ ₂. The node120 is connected for example via an edge 140 to the node 123. The node121 is connected for example via the edge 141 to the node 123.

In this example embodiment, the neural network 100 learns by adjustingthe weighting factors w_(i,j) (weights) of the individual nodes based ontraining data. Possible input values x⁽¹⁾ ₁, x⁽¹⁾ ₂, x⁽¹⁾ ₃ of the inputnodes 120,121,122 can be, for example, the individual field variables{tilde over (E)}_(BC), {tilde over (H)}_(BC), {tilde over (E)}_(i),{tilde over (H)}_(i) and/or examination information UI (if present).

The neural network 100 weights the input values of the input layer 110based on the learning process. The output values of the output layer 112of the neural network 100 preferably correspond to field information FI,based upon which the electrical and/or magnetic field underlying thesignature S may be at least partially suppressed. The output can be madevia an individual or a large number of output node(s) x⁽³⁾ ₁, x⁽³⁾ ₂ inthe output layer 112.

The artificial neural network 100 preferably comprises a hidden layer111, which comprises a large number of nodes x⁽²⁾ ₁, x⁽²⁾ ₂, x⁽²⁾ ₃. Aplurality of hidden layers can be provided, with one hidden layer usingoutput values of a different hidden layer as input values. The nodes ofa hidden layer 111 perform mathematical operations. An output value of anode x⁽²⁾ ₁, x⁽²⁾ ₂, x⁽²⁾ ₃ corresponds to a non-linear function f ofits input values x⁽¹⁾ ₁, x⁽¹⁾ ₂, x⁽¹⁾ ₃ and the weighting factorsw_(i ,j). After receiving input values x⁽¹⁾ ₁, x⁽¹⁾ ₂, x⁽¹⁾ ₃, a nodex⁽²⁾ ₁, x⁽²⁾ ₂, x⁽²⁾ ₃ carries out a summation of a multiplication ofeach input value x⁽¹⁾ ₁, x⁽¹⁾ ₂, x⁽¹⁾ ₃ weighted with the weightingfactors wi,j, as determined by the following function:

x _(j) ^((n+1)) =f(Σ_(i) x _(i) ^((n)) ·w _(i,j) ^((m,n))).

The weighting factor w_(i,j) can be in particular a real number, inparticular lie in the interval from [−1;1] or [0;1]. The weightingfactor w_(i,j) ^((m,n)) designates the weight of the edge between thei^(th) nodes of an m^(th) layer 110,11,112 and a j^(th) node of then^(th) layer 110,111,112.

In particular, an output value of a node x⁽²⁾ ₁, x⁽²⁾ ₂, x⁽²⁾ ₃ isformed as a function f of a node activation, for example a sigmoidalfunction or a linear ramp function. The output values x⁽²⁾ ₁, x⁽²⁾ ₂,x⁽²⁾ ₃ are transferred to the output node(s) 128,129. A summation of aweighted multiplication of each output value x⁽²⁾ ₁, x⁽²⁾ ₂, x⁽²⁾ ₃ iscalculated again as a function of the node activation f, and therewiththe output values x⁽³⁾ ₁, x⁽³⁾ ₂.

The neural network TF shown here is a feedforward neural network inwhich all nodes 111 process the output values of a previous layer in theform of its weighted sum as input values. Of course, inventively, othertypes of neural network can also be used, for example feedback networks,in which an input value of a node can simultaneously also be its outputvalue.

The neural network TF can be trained by way of a method of supervisedlearning in order to provide the field information FI. A known procedureis back propagation, which can be applied to all example embodiments ofthe invention. During training, the neural network TF is applied totraining input data or values and has to generate corresponding, knowntraining output data or values. Mean square errors (MSE) are iterativelycalculated between calculated and expected output values and individualweighting factors are adjusted until the deviation between calculatedand expected output values lies below a predetermined threshold.

For the provision of training data, for example visualization parametersor mapping rules AV can be drawn on, which were created by visualizationexperts for a particular medical context and for particular volume data.In addition, mapping rules AV can be drawn on, which a user N deemedgood in the past or which he has at least not rectified further.

In some implementations a user input can also be received, which refersto the visualization image VB. The user input can comprise, for example,a command to change one or more visualization parameter(s). Furthermore,the user input can comprise an acceptance of the visualization image forthe further workflow. The trained function TF can then be adjustedfurther based upon such a user input by commands for adjustment beingpenalized on further training of the trained function and the acceptanceof a visualization image VB resulting, with further training, in areinforcement.

FIG. 5 shows an embodiment of a system 200 for training or providing thetrained function TF. The system comprises a processor 210, an interface220, a main memory 230, a storage facility 240 and a database 250. Theprocessor 210, the interface 220, the main memory 230 and the storagefacility 240 can be designed as a computer 290. The processor 210controls the operation of the computer 290 on training of the trainedfunction TF. In particular, the processor 210 can be designed in such away that it carries out the method steps represented in FIG. 9. Theinstructions can be stored in the main memory 230 or in the storagefacility 240 and/or be loaded into the main memory 230 when execution ofthe instructions is desired. The storage facility 240 can be designed asa local storage device or a remote storage device, which can be accessedover a network. The method steps represented in FIG. 9 can be defined bycomputer program products, which are stored in the main memory 230and/or the storage facility 240.

The database 250 can be implemented as a Cloud storage device or localstorage device, which is connected to the computer 290 via the wirelessor wired interface 220. The database 250 can, in particular, also bepart of the computer 290. The database 250 serves as an archive for the(training) volume data, training context data and/or associated trainingmapping rules. Furthermore, the database 250 can serve as an archive forone or more trained function(s) TF.

FIG. 6 represents a schematic flowchart of a method for the provision ofa trained function TF for the provision of a mapping rule AV. The orderof the method steps is limited by neither the represented sequence norby the chosen numbering. The order of the steps can thus optionally beinterchanged and individual steps can be omitted.

A first step T10 is directed toward providing a trained function TF. Theprocessor 210 can be provided with the trained function TF by thedatabase 250 via the interface 220. The trained function TF can bepre-trained already, in other words one or more parameter(s) of thetrained function TF have already been adjusted by the described trainingmethod and/or a different training method. Alternatively, the one ormore parameter(s) of the trained function can have not been adjusted byway of training data yet, in particular the one or more parameter(s) canbe pre-allocated by a constant value and/or by a random value. Inparticular, all parameters of the trained function TF can have not beenadjusted by way of training data yet, in particular all parameters canbe pre-allocated by a constant value and/or by a random value.

A second step T20 is directed toward providing training input data.Since, in use, the trained function TF is to qualitatively providemapping rules AV appropriate to the respective context data KD, suitabletraining input data is precisely training volume data and trainingcontext data.

Step T30 is directed toward providing training output data. The trainingoutput data is training mapping rules in this case. The training mappingrules constitute expedient mapping rules AV, which allow a suitablevisualization.

In a next step T40, the training input data, training volume data andtraining context data, therefore, is input into the trained function TF.On this basis, the trained function TF calculates a mapping rule AV,which should enable a suitable visualization of the training volume databased on the training context data.

In a next step T50, the thus calculated mapping rule AV is compared withthe associated training mapping rule. The trained function TF can thenbe adjusted in step T60 based upon this comparison. This can occur, forexample, based upon a cost functional, which penalizes deviations of thecalculated mapping rule AV from the associated training mapping rule.One or more parameter(s) of the trained function TF can then be adjustedin particular such that the cost functional is minimized, for example byway of a back propagation. The cost functional can be based in oneembodiment on a pair-wise difference in control or visualizationparameters of the calculated mapping rule AV and the associated trainingmapping rule, for example on the sum of the deviations squared. In orderto minimize the cost functional the comparison is carried out fordifferent pair-wise sets of calculated mapping rule AV and associatedtraining mapping rule until a local minimum of the cost functional isachieved and the trained function TF is working satisfactorily.

Where it has not yet explicitly occurred but is expedient and within themeaning of the invention, individual example embodiments, individualpartial aspects or features thereof can be combined with each other orreplaced without departing from the scope of the present invention.Advantages of the invention described in relation to one exampleembodiment also apply without explicit mention, where transferable, toother example embodiments.

The following points are likewise part of the disclosure:

-   1. A computer-implemented method for the provision of a    two-dimensional visualization image having a plurality of    visualization pixels for the visualization of a three-dimensional    object represented by volume data for a user, wherein the method has    the following steps:

providing the volume data;

providing context data containing natural language and/or text inrespect of the visualization of the three-dimensional object;

determining a mapping rule based on the volume data and the contextdata, which mapping rule is capable of mapping the volume data onto avisualization image that is suitable when taking into account thecontext data;

mapping the volume data onto the visualization pixels using the mappingrule for the creation of the visualization image; and

providing the visualization image.

-   2. The method as claimed in 1, wherein the mapping of the volume    data takes place by way of a volume rendering method, in particular    using a ray casting method and/or path tracing method.-   3. The method as claimed in one of the preceding points, wherein

the context data comprises information about one or more preference(s)of the user in respect of the visualization of the volume data.

-   4. The method as claimed in one of the preceding points, wherein

the context data has one or more electronic document(s) containing textrelating to the object to be visualized, which documents comprise, inparticular, one or more medical report(s) and/or medical guideline(s).

-   5. The method as claimed in 4, also with the following step

retrieving one or more of the electronic document(s) from a database.

-   6. The method as claimed in one of the preceding points, wherein    determining the mapping indication and/or the mapping rule takes    place with a configuration algorithm, which comprises, in    particular, a computer linguistics algorithm.-   7. The method as claimed in 6, wherein

the configuration algorithm has a trained function.

-   8. The method as claimed in one of the preceding points, wherein

the volume data was generated by a medical imaging method and maps, inparticular, one or more anatomy(anatomies) of a patient.

-   9. The method as claimed in one of the preceding points, wherein the    step of mapping comprises:

actuating an computing facility based on the mapping rule for theprovision of the visualization image.

The example embodiments have been described for illustrative purposes.It will be obvious that the described features, steps and workflow maybe varied in many ways. Such variations are not to be regarded as adeparture from the scope of the present invention, and all suchmodifications as would be obvious to one skilled in the art are intendedto be included within the scope of the following claims.

Of course, the embodiments of the method according to the invention andthe imaging apparatus according to the invention described here shouldbe understood as being example. Therefore, individual embodiments may beexpanded by features of other embodiments. In particular, the sequenceof the method steps of the method according to the invention should beunderstood as being example. The individual steps can also be performedin a different order or overlap partially or completely in terms oftime.

The patent claims of the application are formulation proposals withoutprejudice for obtaining more extensive patent protection. The applicantreserves the right to claim even further combinations of featurespreviously disclosed only in the description and/or drawings.

References back that are used in dependent claims indicate the furtherembodiment of the subject matter of the main claim by way of thefeatures of the respective dependent claim; they should not beunderstood as dispensing with obtaining independent protection of thesubject matter for the combinations of features in the referred-backdependent claims. Furthermore, with regard to interpreting the claims,where a feature is concretized in more specific detail in a subordinateclaim, it should be assumed that such a restriction is not present inthe respective preceding claims.

Since the subject matter of the dependent claims in relation to theprior art on the priority date may form separate and independentinventions, the applicant reserves the right to make them the subjectmatter of independent claims or divisional declarations. They mayfurthermore also contain independent inventions which have aconfiguration that is independent of the subject matters of thepreceding dependent claims.

None of the elements recited in the claims are intended to be ameans-plus-function element within the meaning of 35 U.S.C. § 112(f)unless an element is expressly recited using the phrase “means for” or,in the case of a method claim, using the phrases “operation for” or“step for.”

Example embodiments being thus described, it will be obvious that thesame may be varied in many ways. Such variations are not to be regardedas a departure from the spirit and scope of the present invention, andall such modifications as would be obvious to one skilled in the art areintended to be included within the scope of the following claims.

What is claimed is:
 1. A computer-implemented method for creating atwo-dimensional visualization image including a plurality ofvisualization pixels for visualization of a three-dimensional objectrepresented by volume data for a user, the method comprising: providingcontext data containing at least one of natural language and text forthe visualization of the three-dimensional object; determining a mappingrule based on the volume data and the context data; and mapping thevolume data onto the plurality of visualization pixels using the mappingrule, for creation of the two-dimensional visualization image.
 2. Themethod of claim 1, wherein the volume data includes medical image dataof a patient, and wherein the providing of the context data includes:retrieving patient data of the patient from a database, analyzing textinformation contained in the patient data using a text analysisalgorithm, to produce a text analysis result, and providing the textanalysis result as the context data.
 3. The method of claim 1, whereinthe determining of the mapping rule includes at least one of:establishing a suitable perspective for the two-dimensionalvisualization image; establishing a suitable scene illumination for thetwo-dimensional visualization image; establishing a suitable transferfunction for calculation of visualization pixels of the for thetwo-dimensional visualization image from the volume data; establishing asuitable transparency of one or more regions of the volume data;establishing a suitable cutting plane through the volume data;establishing a suitable region of interest inside the volume data forrepresentation in the two-dimensional visualization image; andestablishing one or more suitable items of additional information foroverlaying in the two-dimensional visualization image.
 4. The method ofclaim 1, wherein the determining of the mapping rule includes: selectinga volume data set for visualization from the volume data based on thecontext data, wherein the two-dimensional visualization image isgenerated based upon the volume data set selected.
 5. The method ofclaim 1, wherein the object includes a plurality of individual objects,represented in the volume data; and the determining of the mapping ruleincludes selection of one or more of the plurality of individual objectsfor representation in the two-dimensional visualization image based onthe context data.
 6. The method of claim 1, wherein the providing of thecontext information comprises: receiving an audio signal containing aspeech input of the user via a detection facility; analyzing the speechinput using a speech analysis algorithm to determine one or more voicecommands of the user; and providing the one or more voice commands ascontext data.
 7. The method of claim 6, wherein the one or more voicecommands are provided after providing the two-dimensional visualizationimage as context data, the method further comprising: adjusting themapping rule based on the context data, to create an adjusted mappingrule; mapping the volume data onto the visualization pixels using theadjusted mapping rule, to create an adjusted visualization image;providing the adjusted visualization image.
 8. The method of claim 1,wherein the context data includes one or more electronic documentscontaining text relating to the object to be visualized.
 9. The methodof claim 1, wherein the determining of the mapping rule comprises:providing a trained function, adjusted to determine a mapping rule basedon context data and volume data, the trained function being configuredto match a visualization image to the context data; and determining themapping rule by applying the trained function to the volume data and thecontext data.
 10. The method of claim 9, wherein the trained functionhas a neural network.
 11. An apparatus for providing a two-dimensionalvisualization image including a plurality of visualization pixels forvisualization of a three-dimensional object represented by volume datafor a user, comprising: an interface, designed to receive the volumedata and at least one of natural language and context data containingtext relating to the visualization of the three-dimensional object; acomputing facility, designed to determine a mapping rule based on thevolume data and the context data; and a visualization module, designedto map the volume data onto the plurality of visualization pixels usingthe mapping rule, to create the two-dimensional visualization image; andprovide the two-dimensional visualization image.
 12. A non-transitorycomputer program product storing a computer program, directly loadableinto a storage device of a computing facility, including programsegments to carry out the method of claim 1 when the program segmentsare run in the computing facility.
 13. A non-transitorycomputer-readable storage medium storing program segments, readable andrunnable by a computing facility, to carry out the method of claim 1when the program segments are run in the computing facility.
 14. Themethod of claim 1, further comprising: providing the two-dimensionalvisualization image.
 15. The method of claim 2, wherein the determiningof the mapping rule includes at least one of: establishing a suitableperspective for the two-dimensional visualization image; establishing asuitable scene illumination for the two-dimensional visualization image;establishing a suitable transfer function for calculation ofvisualization pixels of the for the two-dimensional visualization imagefrom the volume data; establishing a suitable transparency of one ormore regions of the volume data; establishing a suitable cutting planethrough the volume data; establishing a suitable region of interestinside the volume data for representation in the two-dimensionalvisualization image; and establishing one or more suitable items ofadditional information for overlaying in the two-dimensionalvisualization image.
 16. The method of claim 2, wherein the determiningof the mapping rule includes: selecting a volume data set forvisualization from the volume data based on the context data, whereinthe two-dimensional visualization image is generated based upon thevolume data set selected.
 17. The method of claim 2, wherein the objectincludes a plurality of individual objects, represented in the volumedata; and the determining of the mapping rule includes selection of oneor more of the plurality of individual objects for representation in thetwo-dimensional visualization image based on the context data.
 18. Themethod of claim 2, wherein the providing of the context informationcomprises: receiving an audio signal containing a speech input of theuser via a detection facility; analyzing the speech input using a speechanalysis algorithm to determine one or more voice commands of the user;and providing the one or more voice commands as context data.
 19. Themethod of claim 18, wherein the one or more voice commands are providedafter providing the two-dimensional visualization image as context data,the method further comprising: adjusting the mapping rule based on thecontext data, to create an adjusted mapping rule; mapping the volumedata onto the visualization pixels using the adjusted mapping rule, tocreate an adjusted visualization image; providing the adjustedvisualization image.
 20. The method of claim 8, wherein the one or moreelectronic documents include at least one of one or more medical reportsand one or more medical guidelines.
 21. The method of claim 2, whereinthe determining of the mapping rule comprises: providing a trainedfunction, adjusted to determine a mapping rule based on context data andvolume data, the trained function being configured to match avisualization image to the context data; and determining the mappingrule by applying the trained function to the volume data and the contextdata.
 22. The method of claim 9, wherein the trained function has aconvolutional neural network.
 23. A non-transitory computer programproduct storing a computer program, directly loadable into a storagedevice of a computing facility, including program segments to carry outthe method of claim 2 when the program segments are run in the computingfacility.
 24. A non-transitory computer-readable storage medium storingprogram segments, readable and runnable by a computing facility, tocarry out the method of claim 2 when the program segments are run in thecomputing facility.
 25. An apparatus for providing a two-dimensionalvisualization image including a plurality of visualization pixels forvisualization of a three-dimensional object represented by volume datafor a user, comprising: an interface to receive the volume data and atleast one of natural language and context data containing text relatingto the visualization of the three-dimensional object; at least oneprocessor to determine a mapping rule based on the volume data and thecontext data to map the volume data onto the plurality of visualizationpixels using the mapping rule, to create the two-dimensionalvisualization image.
 26. The apparatus of claim 25, further comprising adisplay to display the two-dimensional visualization image.